dynamic search space
Dynamic Design of Machine Learning Pipelines via Metalearning
Alcobaça, Edesio, de Carvalho, André C. P. L. F.
Automated Machine Learning (AutoML) has become an essential tool for democratizing machine learning (ML) by automating key aspects of model selection, hyperparameter tuning, and feature engineering [1, 2]. However, the efficiency of AutoML frameworks remains a significant challenge, as the search for optimal configurations is often computationally expensive [3-5]. Traditional search strategies, such as Random Search (RS) and Bayesian Optimization (BO), indiscriminately explore large search spaces, resulting in high resource consumption [3, 6, 7]. To address this challenge, we propose a metalearning approach that dynamically designs search spaces for an AutoML solution, reducing computational costs while maintaining competitive predictive performance. The proposed method leverages historical metaknowledge to identify and prioritize promising regions of the search space, enabling more efficient optimization. By predicting the performance of preprocessor-classifier combinations, a meta-model, induced using metalearning, can provide a warm-start advantage, accelerating the AutoML search process. This study evaluates the effectiveness of the proposed approach through an extensive set of experiments, analyzing both computational efficiency and predictive performance. According to the experimental results, the dynamically generated search spaces significantly reduce runtime, while maintaining high-quality solutions. In particular, the RS-mtl-95 configuration achieved an 89% reduction in runtime without compromising predictive performance.
Optimizing Neural Architecture Search using Limited GPU Time in a Dynamic Search Space: A Gene Expression Programming Approach
Alves, Jeovane Honorio, de Oliveira, Lucas Ferrari
Efficient identification of people and objects, segmentation of regions of interest and extraction of relevant data in images, texts, audios and videos are evolving considerably in these past years, which deep learning methods, combined with recent improvements in computational resources, contributed greatly for this achievement. Although its outstanding potential, development of efficient architectures and modules requires expert knowledge and amount of resource time available. In this paper, we propose an evolutionary-based neural architecture search approach for efficient discovery of convolutional models in a dynamic search space, within only 24 GPU hours. With its efficient search environment and phenotype representation, Gene Expression Programming is adapted for network's cell generation. Despite having limited GPU resource time and broad search space, our proposal achieved similar state-of-the-art to manually-designed convolutional networks and also NAS-generated ones, even beating similar constrained evolutionary-based NAS works. The best cells in different runs achieved stable results, with a mean error of 2.82% in CIFAR-10 dataset (which the best model achieved an error of 2.67%) and 18.83% for CIFAR-100 (best model with 18.16%). For ImageNet in the mobile setting, our best model achieved top-1 and top-5 errors of 29.51% and 10.37%, respectively. Although evolutionary-based NAS works were reported to require a considerable amount of GPU time for architecture search, our approach obtained promising results in little time, encouraging further experiments in evolutionary-based NAS, for search and network representation improvements.